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Abstract 

Virginia's assessment of student learning outcomes has been lauded by national organizations 
for its respect of institutional autonomy while providing meaningful information on student learning 
outcomes. Virginia recently implemented a process by which each institution’s plan to assess student 
learning outcomes are evaluated by peer institutions. The application of peer review to plans for assess¬ 
ment is described in greater detail and critiqued using the theoretical lens afforded by organizational 
learning. The article concludes with discussion and recommendations for the improvement of the peer 
review process as it applies to assessment. 

Introduction 

Virginia’s assessment of student learning outcomes has been lauded by national organizations 
for its respect of institutional autonomy while providing meaningful information on student learning 
outcomes (Epstein, 2005). Virginia recently implemented a process by which each institution’s plans to 
assess student learning outcomes are evaluated by peer institutions. The application of peer review to 
plans to assess student competency is described in greater detail and critiqued using the theoretical lens 
afforded by organizational learning. The article concludes with discussion and recommendations for the 
improvement of the peer review process as it applies to assessment. 

In order to best understand the application of peer review to the process of competency assess¬ 
ment in Virginia, it is necessary to begin by describing the process of competency assessment in Virginia 
and the recent addition of peer review to this process. Following a description of assessment of student 
learning in Virginia, a brief discussion of organizational learning is conducted that includes an operation¬ 
al definition of the term and a tentative description of the organization in question. Next, the use of peer 
review in higher education is surveyed with particular attention paid to the benefits and shortcomings of 
the process. Finally, a discussion of improvements to peer reviewed student outcomes assessment is under¬ 
taken and recommendations are made using the theoretical constructs provided by organizational learning. 

It should be noted from the onset that the topics of organizational learning, peer review, and 
student competency assessment are far broader than the limitations of this article. This article seeks to 
illuminate the areas of critical overlap between organizational learning, the use of a peer review process, 
and the assessment of student learning in an attempt to improve the assessment process. 

Competency Assessment in Virginia 

In 1998, the Governor of Virginia charged a Blue Ribbon Commission with evaluating the 
needs and goals of higher education in Virginia for the 21st century. Among the specific charges to the 
Commission, the Governor requested that the Commission, “advise the Governor on how the institu¬ 
tions, administrators, and faculty that comprise Virginia’s system of higher education can be made more 
accountable to their stockholders (the taxpayers, the parents, and the private contributors who finance the 
system) for the quality of the academic content and the outcomes accomplished through the investment 
of public funds.” (Executive Order 1,1998). The Commission concluded that evidence of high quality 
outputs is essential in assuring stockholders that the substantial investment made by the Commonwealth 
in higher education is producing results (Governor’s Blue Ribbon Commission on Higher Education, 

2000). In order to provide this assurance, the Commission identified six areas of core competency—areas 
of knowledge and skill that supersede majors, disciplines, and institutional missions— recommending 
that these areas be assessed regularly and the results of such assessments be shared with the public. The 
core competencies identified by the Commission included written communication, mathematical analy¬ 
sis, scientific literacy, critical thinking, oral communication, and technology. 

The Code of Virginia was subsequently amended such that the State Council of Higher Edu¬ 
cation for Virginia (SCHEV), Virginia’s coordinating body for higher education, was charged with 
“developing] in cooperation with institutions of higher education guidelines for the assessment of 
student achievement” (Code of Virginia, 2000). Biennially, and starting in 2001, each public four-year in¬ 
stitution of higher education in the Commonwealth submitted plans to assess competency in two speci- 
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fied areas. The first of three rounds of assessment required institutions to submit plans to assess student 
competency in written communication and technology/information literacy to SCHEV for approval one 
year prior to submitting results. SCHEV staff reviewed the plans in light of each institution’s mission and 
provided feedback with notification of approval. The same process was used in 2003 when plans to assess 
student competency in scientific reasoning and quantitative reasoning (adapted from the Blue Ribbon 
Commission’s recommendation to assess scientific literacy and mathematical analysis, respectively) were 
submitted to SCHEV staff for review and approval. 

In 2005, SCHEV staff instituted a process of peer review, by which each institution’s plans to 
assess critical thinking and oral communication were shared with two other institutions in the Common¬ 
wealth for the purpose of (a) providing each institution with expert feedback; and (b) initiating inter-in¬ 
stitutional communication on topic of competency assessment plans for the purpose of providing mutual 
benefit to reviewer and the reviewed while spurring creative approaches to assessment. Some six months 
before the peer review process began, SCHEV staff solicited the opinion of assessment professionals 
regarding the use of a peer review process in place of a review of institutional plans to assess student 
competency conducted exclusively by SCHEV staff. General support for such a process was expressed by 
assessment professionals. 

In the spring of 2005, each of Virginia’s 15 public four-year institutions submitted plans ap¬ 
proved by its chief academic officer to assess student competency in critical thinking and oral communi¬ 
cation. Plans included a definition of the competency used by the institution, criteria and standards for 
determining competency, and a methodology for scoring and deeming students competent. Each insti¬ 
tution’s designated assessment coordinator was then provided with plans from two other institutions in 
the Commonwealth: one institution that was of the same Carnegie classification and one that was from 
a different Carnegie classification (Carnegie Classification, 2000). Each institutional representative was 
encouraged to form a committee of knowledgeable staff from his or her institution to review the four as¬ 
signed competency assessment plans (two competencies from two universities) and provide written feed¬ 
back in accordance with a set of suggested components. Examples of the suggested components provided 
to referees for the purpose of evaluating an institution’s plans to assess competency read, “adequacy of 
criteria and standards for determining competency” and “appropriateness of competency to the mission, 
goals, and objectives of the institution” (SCHEV memo to assessment officers, 2005). 

Peer reviews were collected and compiled by SCHEV staff who acted as editors of the peer 
review comments much in the way that a journal editor does of peer reviewed publication. SCHEV staff 
read each plan and each review before sharing anonymous feedback with each institution. The feed¬ 
back outlined the concerns and praise raised by referees in addition to concerns and praise generated by 
SCHEV staff. The process resulted in a single blind review, in that the reviewers were explicitly notified 
of the institutions they were reviewing, while recipients of review were not notified of the institutions 
that conducted the review. A double blind process was not possible given that referees were required to 
compare each institution’s plans to assess competency with the institution’s mission, goals, and objectives. 
Each institution received peer review comments within 45 days of submitting plans to assess competency 
and one full year before the results from the competency assessments were due. 

Organizational Learning 

Organizational theory is the study of how “groups and individuals behave in varying organiza¬ 
tional structures and circumstances” (Shafritz & Ott, 2001, p.l). The study of organizational theory helps 
those that manage higher education to understand complex concepts (Berger, 2000; Birnbaum, 1988). 
Further, the application of organizational theory to complex concepts allows for a more complete un¬ 
derstanding of the concept and the ability to take wellinformed action (Berger 2000, Birnbaum, 1988; 
Bolman & Deal, 2003; Morgan, 1997). The term organizational theory refers broadly to the theoretical 
frames and perspectives applied to the study of organizational behavior (Morgan, 1997; Shafritz & Ott, 
2001). In an effort to better understand the processes of peer reviewed student learning assessment and 
make recommendations for its improvement, the theoretical frame of organizational learning will be ap¬ 
plied to peer review and student competency assessment. 

Organizational learning is, in itself, an umbrella term for a set of organizational theories that 
ascribe learning characteristics to organizations (Morgan, 1997). Taxonomists of organizational learning 
have classified its theories in a number of ways (Argyris & Schon, 1996; Dierkes, Berthoin Antal, Child, 
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&Nonaka, 2001; Morgan, 1997). Dierkes et al. (2001) distinguish between theories that speak to the 
creation of new knowledge and those that speak to the sharing, using, and storing of knowledge. Morgan 
(1997) distinguishes between theories associated with acquiring, processing, and using knowledge and 
those associated with storing and accessing knowledge. Argyris and Schon (1996) distinguish between a 
practically-oriented branch of organizational learning and a scholarly-oriented branch of organizational 
learning that is distant from practice. The immediate discussion of organizational learning, as it applies 
to the peer review process of plans to assess student learning outcomes, will begin by focusing on how 
knowledge is practically shared, processed, and used. The discussion will then migrate to the formation of 
new knowledge. 

In order to best understand how organizational learning will be applied to the concept of peer 
review and ultimately the assessment of student learning, it is necessary and appropriate to provide a 
functional definition of organizational learning that narrowly describes its use in this process by which 
organizations share, process, and use knowledge by scanning the environment, comparing what is ob¬ 
served to operating norms, and correcting accordingly. The act of scanning, comparing, and correcting 
reflects a single loop learning orientation. A double loop learning orientation to organizational learning, 
in which existing norms are questioned, will be introduced in the discussion section. 

Community of Practice 

Given the existence of learning organizations, it must be established that an organization among 
assessment professionals in the Commonwealth of Virginia exists if the theory is to be applied to the dis¬ 
cussion at hand. Wenger and Snyder (2001) coined the phrase community of practice to describe people 
informally bound by shared expertise who, in turn, share knowledge beyond the traditional boundaries 
of their formal organization for the purpose of creatively approaching shared problems. Looking at this 
definition in parts, the argument can be made that assessment professionals at Virginia’s public institu¬ 
tions of higher education (a) contain shared expertise; (b) are formally bound to their own institution; 
and (c) are encouraged to share knowledge with colleagues for the purpose of approaching shared prob¬ 
lems through a process of peer review. 

The argument that assessment professionals in Virginia constitute an informal organization is 
not without its flaws. First and foremost, the association of assessment professionals formed through peer 
review is not a free association, as the members have been, to some degree, compelled to participate. But 
the very nature of organization results in variation in participation levels and willingness to participate 
among members (Bolman & Deal, 2003; Morgan, 1997). Second, members of the loosely coupled orga¬ 
nization of peer reviewers are not bound to creatively approach problems. Yet the most recent assessment 
of the critical thinking and oral communication core competencies were thought to be the most difficult 
competencies to assess of the six that Virginia has identified. Peer review, as a mechanism for sharing 
information, was intended to heighten creative problem solving with regard to the development of plans 
to assess critical thinking and oral communication. 

Peer Review 

Peer review is a widely practiced form of certifying quality in higher education. Peer review has 
been described as a formative evaluation process in which participants work collaboratively to strengthen 
a product (Keig & Waggoner, 1994). Common uses of peer review in higher education include the 
awarding of research funds, evaluating academic publications, reviewing faculty performance for tenure 
and promotion, and granting regional and disciplinary accreditation. Peer review is generally said to 
encourage critical examination, promote the exchange of ideas, reduce non-academic interference, guide 
academic discourse, and reinforce academic values (Berkencotter, 1995). In addition to its benefits, peer 
review has been criticized for suppressing innovation, promoting cliques, and providing irreproducible 
results (Harnard, 1982; Peters &Ceci, 1982; Rothwell &Martyn, 2000). Focusing on the benefits and 
shortcomings of peer review, it is important to draw connections between peer review and organizational 
learning where such connections exist. 

An alignment may be identified between the constructs of organizational learning defined previ¬ 
ously and the benefits and shortcomings of peer review. It is this alignment that permits for a better 
understanding of peer review, an improvement of its practice, and an improvement in the assessment of 
student learning. Peer review, like the single loop learning process of organizational learning, assumes 
the existence of norms by which a peer’s work may be judged. Through critical examination, norms are 
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used to compare a peer’s work to accepted practices. If a peer’s work deviates significantly from accepted 
norms, then an attempt to correct will likely occur. Harnard (1982) refers to this action of peer review as 
“selfcorrective”, in the sense that experts in the disciple are maintaining the discipline’s accepted norms. 
The same terminology is used by organizational learning scholar Morgan (1997) to describe single loop 
organizational learning. Peer review, as a form of organizational learning, uses norms to guide a self¬ 
corrective process. 

Self-correction exposes peer review and single loop learning to a major criticism. Single loop 
learning is, by its very nature, a perpetuation of the norms of the organization. Like peer review, ideas 
that deviate from the norms of the organization are corrected. The perpetuation of norms can lead to the 
suppression of innovation in peer review and the inability to adapt and change in an organization. Up until 
this point, the discussion of organizational learning has been confined to the single loop learning process. 

Discussion 

In order to overcome the impediment to organizational learning that is created by the sup¬ 
pression of innovation, a new view of organizational learning must be adopted. Double loop learning 
encourages participants in organizational learning to challenge the norms that guide corrective action 
(Morgan, 1997). The cliche “thinking outside the box” is often used to describe the process of challeng¬ 
ing existing norms. When faced with an idea or practice that deviates from existing norms, double loop 
learning encourages the learner to challenge the norms rather than immediately discard the innovation, 
as single loop learning would dictate. For example, a method of assessment that does not resemble the 
status quo may be discouraged by peer review that is guided by single loop learning. The introduction of 
double loop learning permits the reviewer to challenge the status quo and further explore the innovative 
technique through a dialogue with its creator. 

Perhaps the strongest bond that exists between peer review and organizational learning is that 
which is exposed only when double loop learning is introduced. Organizational learning has the po¬ 
tential to promote the mutual sharing of knowledge (Argyris & Schon, 1996). Double loop learning 
furthers this sharing of knowledge by permitting for the creation of new knowledge. Peer review also 
seeks mutual benefit through the sharing of information. Peer reviewers and those receiving review can 
benefit from the exchange afforded by the peer review process. If the mutual benefits of peer review are 
to be realized, an iterative process must be instituted (Rubin, 1982). The peer review of plans to assess 
competency in critical thinking an oral communication fell short of a process that was truly iterative, in 
that reviewers were only afforded a one-time, one-way opportunity to address an institution’s plans to 
assess competency. An iterative process would provide the opportunity to review, rebut, and revise in a 
cyclical method until a suitable finished product is reached. 

The following suggestions are made in an effort to improve the peer review process as it applies 
to the assessment of student learning. First, double loop learning requires that the norms used to guide 
decision making and corrective action (i.e. the norms that guide an organization) must be continuously 
challenged (Morgan, 1997). In efforts to advance new ideas and promote innovation in student out¬ 
comes assessment, operating norms and assumptions must be confronted, ultimately resulting in their 
affirmation or their dismissal and replacement. Morgan suggests that organizations interested in fully 
developing double loop learning strategies (a) anticipate change; (b) develop capabilities for questioning 
operating norms; and (c) foster emergent organization. 

Second, a balance must be struck between innovation and regulation. Steps must be taken to 
foster innovation and the free exchange of ideas, as is required in a community of practice, while still 
engaging in a process that is ultimately regulated by state code. Bureaucracies are widely criticized for 
stifling innovation. At the same time, a level of consistency and order must be achieved to comply with 
the intent of the policy that aims to provide substantive information on the quality of student learning. 
Goodsell (1994) notes that the ability to stabilize and provide predictability are among bureaucracy’s 
greatest virtues. In all, forces for innovation must confront forces for stabilization and predictability. 

Finally, in order to achieve an environment in which innovation is fostered and properly bal¬ 
anced, communication must be optimized (Berkencotter, 1995). Managers of information must act 
diligently and deliberately to establish networks that support collegial interaction. Organizers of peer 
review should consider how a truly iterative process may be implemented if the full benefits of peer 
review are to be realized. Further, members of the organization must be willing to engage in ongoing 
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and substantive discussions on what constitutes good assessment. The original intent of using peer review 
in evaluating student competency assessment plans was to (a) provide expert feedback, and (b) encour¬ 
age inter-institutional communication for the purpose of providing mutual benefit to reviewer and the 
reviewed while spurring creative approaches to assessment. Given that the plans reviewed have yet to be 
fully implemented, it is too soon to determine if both aims have been achieved. Evaluation of the first 
aim will require the completion of the assessment cycle to determine if the expertise of peer comments 
provided greater value than those of SCHEV staff. The second aim will also require the sort of reflection 
that is best accrued with time. Ultimately, the literature suggests that a double loop learning process will 
contribute to mutual learning and the use of innovative assessment techniques, if the proper conditions are 
established. Participants and organizers (i.e. the reviewed, reviewers, and editors) must engage in an ongo¬ 
ing discussion that seeks to clarify good assessment without suppressing innovation. 

Author note: Craig Elerndon serves as Associate for Academic Affairs and Research Policy 
Analyst for the State Council of Higher Education for Virginia. In addition, he is a doctoral 
candidate in Virginia Tech’s Educational Policy and Leadership Studies program. 
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